Unsupervised Neural Network Learning for Blind Sources Separation

Review of Independent Component Analyses (ICA) and Blind Sources Separation (BSS) employing in terms of unsupervised neural networks technology are given. For example, imagery features occurring in human visual systems are the continuing reduction of redundancy towards the “sparse edge maps”. Different edge features are mathematically equivalent to be “ones among zeros”. When edges are multiplying together as the vector inner product resulted in almost zero, namely pseudo-orthogonal ICA. This fact has been derived from the first principle of artificial neural networks using the maximum entropy information-theoretical formalism by Bell & Sejnowski in 1996. We explore the blind de-mixing condition for more than two objects using two sensor measurement. We design two smart cameras with short term working memory to do better image de-mixing of more than two objects. We consider channel communication application that we can efficiently mix four images using matrices [A,,] and [A,] to send through two channels. 1. Unsupervised Learning and Human Vision Principles: Independence, Orthogonal@ and Sparseness An unsupervised learning strategy of Artificial Neural Network (ANN) is to change the weight matrix (W] of ANN to sieve and squeeze in parallel all the useful information from the time series of input vector x(t) until the output vector u(t) = PI x(t) contains no more useful information in the sense at maximum entropy H(u) shown in Fig. 1. In other words, all the good information is already kept in the memory matrix [WJ, which turns out to be undoing the image formation, the inverse operation of mixing lights, in short, de-mixing described mathematically as follows. At this point, it is appropriate to make a comment. This unsupervised strategy is different to the traditional supervised learning, because one can no longer assume any specific output goal for exemplar inputs. After the input having been sieved out all the useful information, the most natural output is one that “garbage-output” by the strict definition of no supervision. Such an unsupervised learning strategy of neurocomputers is described by “data-in & garbage-out” as opposed to the usual motto-“garbage-in & garbage-out” in traditional computers. This new paradigm is useful for solving the matrix inversion [A]-’ statistically which underlies the Independent Component Analyses (ICA) mathematically as follows: u(t) = [W] x(t)=[W] [A] s(t),where t stands for both time of signal or the scanning order of pixels. If the learning of weight matrix [W] can achieves the maximum entropy H(u) of the output u or the linear slope portion of the maximum entropy sigmoidal neuron output H(y)=H(o(u))=H(u) which implies that al1 nth moments of the ANN output components u={u,, u$ of two sensor neurons are independent in terms of the normalized statistical histograms p(u) defmed as: < U’l >= I IrJ’p(u)du. Specifically, the whitening of the second moment of the output shows: ~(+‘(t)> = [W][A] [A]‘[W]‘= [I] known as Oja’s sphering is equivalent to PV = PI-’ provide that statistical de-correlation of sources = [I] is true (if not, pre-whitening filter [Wz] = “’ is often used by Bell-Sejnowski et. al. The fourth cumulant, the Kurtosis K(u), is often used by Helsinki’s group to seek the statistical matrix inversion. K(u) = 3 ()’ in terms of a single weight vector update: dw/dt = dx/dw. The other weight vectors are found by the projection pursuits. Unsupervised Learning